Reducing precision in representing numerical values for mathematical computations may be beneficial in certain circumstances. For example, 32-bit floating-point values can be replaced with less precise 16-bit floating-point values to reduce circuitry, power, and bandwidth for training of neural networks. The reduced precision values require only half the bandwidth during data transmission and less than half the die area and power consumption compared with using full precision values. However, the range of values that can be represented using a 16-bit floating-point format is much smaller than the range of values that can be represented using a 32-bit floating-point format. Numbers smaller than the smallest value represented by the 16-bit floating-point format are lost (i.e., turned to zero).
Stochastic rounding is a method that is conventionally used to extend numerical range. With traditional rounding, numbers are deterministically rounded up or down, for example values between 0.5 and 1 are rounded up to 1, values below 0.5 are always rounded down to 0. With stochastic rounding, the rounding is instead probabilistic. Specifically, 0.5% has 50% odds of being rounded up to 1 and 50% odds of being rounded down to 0, while 0.1 has 10% odds of being rounded up to 1 and 90% odds of being rounded down to 0.
With stochastic rounding, an individual rounding event can actually introduce more error, but on average over a long sequence of accumulations, the result will have less error. For example, when accumulating 1000 numbers with value 0.1 with traditional rounding after each number is accumulated, the result will be zero, whereas with stochastic rounding the result should be closer to the correct answer of 100.
Thus while stochastic rounding is preferable when applying an accumulate and round operation over a long series of numerical values, the challenge is implementing the random rounding behavior of stochastic rounding efficiently in software or hardware. One method could be to generate a random number for each rounding operation, but that is extremely expensive in terms of additional circuitry and/or power consumption. There is a need for addressing these issues and/or other issues associated with the prior art.